System and method for storing data

ABSTRACT

A method for provisioning a volume of data is disclosed. The method involves identifying a set of rules associated with the volume of data, wherein the set of rules includes at least two conflicting rules, prioritizing the at least two conflicting rules, placing the volume of data on a first computer system according to the prioritization, collecting performance metrics corresponding to the placed volume of data, computing a score based on the collected performance metrics, determining if the computed score is acceptable, determining that the placement is successful if the computer score is determined to be acceptable, and changing the prioritization of the at least two conflicting rules and placing the volume of data on a second computer system according to the changed prioritization if the score is determined to be unacceptable.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.15/814,022, filed Nov. 15, 2017, now U.S. Pat. No. 10,691,350, issued onJun. 23, 2020, which is entitled to the benefit of provisional U.S.Patent Application Ser. No. 62/422,366, filed Nov. 15, 2016, entitled“METHOD AND SYSTEM FOR MANAGING THE OPERATION OF A COMPUTER SYSTEM SUCHAS A STORAGE SYSTEM,” which is incorporated by reference herein.

FIELD OF THE INVENTION

The invention relates generally to storage systems for storing digitaldata, and, more specifically to computer-based techniques for storingdigital data in a volume of a storage system.

BACKGROUND

In the field of distributed computer systems, resources are madeavailable for use by computer applications. Computer applications mayrequire different computer resource types, such as central processingunit (CPU), memory, network, and data storage. Resources from multiplecomputer systems may be aggregated into one or more resource pools.Resource requests may include multiple resource types.

Selection of resources to satisfy resource requests is made by ascheduler. The scheduler makes decisions based on algorithms or rules.Typically, there is no specific format required for the rules thescheduler uses to make decisions.

Where resource schedulers exist currently, the rules they use arepredefined to select a specific behavior or algorithm. This has thedisadvantage of being difficult to specify or change schedulingbehavior, and difficult to specify or change scheduling behavior foronly some types of resources.

Existing distributed computer systems require that the requestor specifyfeatures or attributes that the required resource must exhibit. Forexample, if an encrypted data volume was required by the requestor, thendetails about the encryption cipher and the encryption key would need tobe provided by the requestor. As resources typically have multiplefeatures and attributes, their specification can be cumbersome anddifficult to manage in large environments, often leading to inconsistentconfiguration and ultimately, incorrect behavior.

SUMMARY OF THE INVENTION

A method for provisioning a volume of data is disclosed. The methodinvolves identifying a set of rules associated with the volume of data,wherein the set of rules includes at least two conflicting rules,prioritizing the at least two conflicting rules, placing the volume ofdata on a first computer system according to the prioritization,collecting performance metrics corresponding to the placed volume ofdata, computing a score based on the collected performance metrics,determining if the computed score is acceptable, determining that theplacement is successful if the computer score is determined to beacceptable, and changing the prioritization of the at least twoconflicting rules and placing the volume of data on a second computersystem according to the changed prioritization if the score isdetermined to be unacceptable.

In an embodiment, the volume of data is placed on the second computersystem only during hours when the second computer system is not used.

In an embodiment, the volume of data is placed on the second computersystem during a period of time in which activity is relatively low.

Other aspects and advantages of embodiments of the present inventionwill become apparent from the following detailed description, taken inconjunction with the accompanying drawings, illustrated by way ofexample of the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates computer resources that can be scheduled to provideservices.

FIG. 2 illustrates an example embodiment of the overall design of therules engine and how the rules engine integrates with the scheduler.

FIG. 3 illustrates an embodiment of the rules engine and scheduler whenused to provision data storage.

FIG. 4 is a flow diagram that illustrates how different inputs areprocessed when adding new rules.

FIG. 5 is a flow diagram that illustrates a bootstrap process of therules engine and an event loop that performs continuous updates based onstate changes.

FIG. 6 is a flow diagram that illustrates an example of how the rulesengine resolves dependencies.

FIG. 7 is a flow diagram that illustrates an example of the environmentscanning submodule of the rules engine.

FIG. 8 illustrates an example of how the rules engine works in a clusterof six computers.

FIG. 9 illustrates an example of how multi-dimensional rules influencedata volume provisioning and continuously work on enforcing rules.

FIG. 10 depicts a computer that includes a processor, memory, and acommunications interface.

Throughout the description, similar reference numbers may be used toidentify similar elements. Additionally, in some cases, referencenumbers are not repeated in each figure in order to preserve the clarityand avoid cluttering of the figures.

DETAILED DESCRIPTION

It will be readily understood that the components of the embodiments asgenerally described herein and illustrated in the appended figures couldbe arranged and designed in a wide variety of different configurations.Thus, the following more detailed description of various embodiments, asrepresented in the figures, is not intended to limit the scope of thepresent disclosure, but is merely representative of various embodiments.While the various aspects of the embodiments are presented in drawings,the drawings are not necessarily drawn to scale unless specificallyindicated.

The described embodiments are to be considered in all respects only asillustrative and not restrictive. The scope of the invention is,therefore, indicated by the appended claims rather than by this detaileddescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

Reference throughout this specification to features, advantages, orsimilar language does not imply that all of the features and advantagesthat may be realized with the present invention should be or are in anysingle embodiment. Rather, language referring to the features andadvantages is understood to mean that a specific feature, advantage, orcharacteristic described in connection with an embodiment is included inat least one embodiment. Thus, discussions of the features andadvantages, and similar language, throughout this specification may, butdo not necessarily, refer to the same embodiment.

Furthermore, the described features, advantages, and characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. One skilled in the relevant art will recognize, in light ofthe description herein, that the invention can be practiced without oneor more of the specific features or advantages of a particularembodiment. In other instances, additional features and advantages maybe recognized in certain embodiments that may not be present in allembodiments of the invention.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the indicatedembodiment is included in at least one embodiment. Thus, the phrases “inone embodiment,” “in an embodiment,” and similar language throughoutthis specification may, but do not necessarily, all refer to the sameembodiment.

The present invention relates generally to computer systems and methodsfor optimizing resource usage in distributed computer systems. Thepresent invention may relate to methods for specifying rules forresource selection and feature enablement within computer systems havingmultiple resources of the same and differing types.

In an embodiment, the present application provides systems and methodsfor applying high-level business objectives to computer systems bytranslation of the objectives into rules, which on receipt of anexternal event, are then evaluated to produce system tasks required forimplementation on relevant computer systems.

With the translation mechanism, the person or entity that specifies thehigh-level business objectives does not need to understand technicalimplementation details. Conversely, the computer system does not need tounderstand business objectives, rather only the tasks that must beperformed to satisfy the objective requirements.

In an embodiment, the translation mechanism also allows for automatedapplication of business objectives by computer systems.

Business objectives, objective translation rules, and implementationtasks can all be defined in external systems.

In an embodiment, a rulebase stores the rules generated by the businessobjective translation. The rules define what must or must not happen tosatisfy the objective, and to what types of computing object the ruleshould apply to.

In one embodiment, a database of storage implementation rules thatallows the rules engine to evaluate and codify business rules in termsof technical implementation operators for a storage system is disclosed.The database may include functionality of the storage system and mayallow the scheduler to manage the data plane of a storage system. Thelist of technical implementation operators includes but is not limitedto: the provision of a volume including the attributes of the volumesuch as the name and size of the volume of data; the data protectionattributes of a volume such as the utilization of one or more encodingmethods that add redundancy to the data within a volume, or encode datain a format where the data is recoverable if a given fraction of thedata is no longer accessible given the remaining fraction of the data;the data availability attributes of a volume such as the utilization ofone or more replication methods where a copy of the data is maintainedon other computers or systems other than the system holding the originaldata; the data availability attributes of a volume such as the servicedefinitions of the replication methods available to a storage systemincluding, for example, methods which may replicate data in real timeensuring that a copy of the data is always consistent with the sourcedata as well as methods where data in a copy is only eventuallyconsistent; the data security attributes of a volume such as the use ofcryptographic methods which are implemented to scramble the data with akey such that the data is only later accessible by decoding with thesame key; the data security attributes of a volume such as the use ofaccess control mechanisms to restrict access to the volume to specificcomputer users or specific computer systems; the data reductionattributes of a volume such as the use of data compression methods toreduce the size of the data that needs to be persisted; the datareduction attributes of a volume such as the use of the datadeduplication methods to remove duplicate copies of the data; the datacaching attributes of a volume where commonly used data within thevolume is retained in a faster medium, for example, computer systemmemory or other fast storage medium, such that access to the data isfaster than would be possible from the storage medium where the data ispersisted; the methods and technical systems that are available toaccess a given volume including, for example, the formats and protocolsused to access a volume through a storage system; and the attributes ofdifferent classes of storage media that are available in a storagesystem, including attributes such as the speed of the media, the formatand access protocol to the media, the durability and expected lifetimeof the media, and any constraints that may apply when accessing themedia.

In addition, the database may include links of business operators to thetechnical operators of the storage system such that the rules engine canparse the business operators and identify the appropriate attributes andactions to be taken by the storage system. Examples of links betweenbusiness operators to technical operators may include but are notlimited to:

“secure” to identify cryptographic methods;

“highly available” to identify replication methods;

“immediately available” to identify synchronous or highly consistentreplication;

“allow access” to identify security controls for access controlmechanisms;

“fast” to identify the caching methods used or the speed of the diskmedia where the volume is placed; and

“archive” to identify the type of disk media to select.

In another embodiment, other domain specific databases can be added bythe operator of the computer or storage system to allow the rules engineto process parameters for system functionality or computer systemresources that have not yet been defined. This allows the technique tobe extensible to cover the development of new technologies that have notbeen released as products yet.

In an embodiment, external events such as resource requests andenvironmental changes are received by a scheduler and a rules engine.The scheduler is responsible for assigning computer resources once therules engine has determined what, if anything, must be changed, andaccording to constraints set by the rules engine. The scheduler willattempt to make best use of computer resources according to placementrules set by the rules engine.

In an embodiment, the scheduler creates implementation plans for changesthat the rules engine determines must be applied to the environment.Implementation plans are comprised of a set of tasks that must beexecuted on specific computer systems. The implementation plans optimizetask execution order to manage dependencies and to allow tasks to run inparallel where possible.

In an embodiment, the rules engine evaluates system inputs such ascomputer resource requests and environmental factors such as availablecomputer hardware resources. The rules engine is constantly evaluatinginputs and can make decisions based on the translated businessobjectives without operator intervention. The decisions the rules enginetakes may be accompanied by constraints that must be satisfied in orderfor the decision to be implemented successfully.

In an embodiment, the rules engine is dynamic and can adapt a runningenvironment based on new or modified business objectives.

In an embodiment, the rules engine can adapt a running environment basedon changes to environmental factors such as failing computer componentsor systems.

In an embodiment, the rules engine can optimize the environment based onrate of change, such as by pre-expansion of computing capacity whengrowth thresholds have been exceeded. Thresholds can be definedstatically or dynamically based on rate of change.

In an embodiment, the rules engine can also optimize the environmentbased on resource usage. This includes both expansion and reduction ofcomputing capacity based on observed usage.

In an embodiment, the rules engine integrates with third party computersystems or computer code at runtime to make decisions based on dataexternal to the system.

In an embodiment, the scheduler integrates with third party computersystems or computer code at runtime to make decisions based on dataexternal to the system or to notify external systems of progress. Thismay include calling external systems to retrieve resource utilizationstatistics that may influence scheduling decisions or change managementsystems to notify on change completion.

In an embodiment, computer objects are assigned metadata that can beused by the rules engine and the scheduler for making decisions.

Computer Resources

FIG. 1 illustrates computer resources that can be organized to provideservices. A computer system 002 has a plurality of resources, includingbut not limited to CPU 006, memory 007, network 008, and storage. CPU isthe processing capacity defined by the number and speed of the computersystem's central processing unit(s). A plurality of computer systems canbe grouped or organized into clusters 003. A plurality of computersystems or clusters can be organized into a datacenter 001. Datacenterscan be owned and operated by the operator of the computer system, or maybe operated as a service by a 3rd-party vendor. Within a datacenter, adatacenter operator can organize one or more resources to createservices 004, 005, where a service provides a specified function basedon an operator defined specification. The services may be furthergrouped and treated as functionally equivalent to an individual computersystem's resources when being processed by the scheduler.

Rules Engine

FIG. 2 illustrates the overall design of a system that incorporates therules engine. In an embodiment, the rules engine 107 runs on a singlecomputer system. There is also a single scheduler 110 running. Both therules engine and the scheduler receive state 108 changes and events 109.

State 108 describes the current system environment, or changes to thesystem environment. The system state may be used by the rules engine orscheduler to trigger resource allocation changes. Examples of systemstate changes include the addition of new system resources, updates tosystem resource usage, or failed system resources.

Events 109 may be resource requests, such as to run a workload, or toprovision data storage. Events may also be notifications about changeswithin the system, such as the creation, update or deletion of a rule inthe rulebase. Events may be user-generated or computer-generated.

The rules engine 107 evaluates events and state changes recursivelyagainst the rulebase 106.

The rules engine 107 may send events and state changes to externalsystems for processing 120. This gives the advantage of allowing therules engine to process complex logic without requiring the logic to beheld within the computer code of the rules engine or within the rules inthe rulebase. In addition, this allows the rules engine system to beextensible without modification of the rules engine.

If the rules engine 107 determines that a change to the existing state108 is required, the rules engine notifies the scheduler 110 to make thechange.

In this embodiment, the scheduler 110 is designed to run on a singlecomputer in a distributed system. The scheduler receives resourcerequests as events or by the rules engine. Resource requests include theattributes and constraints that must be satisfied in order to servicethe request. The scheduler is responsible for making resourceallocations that make the most efficient use of the resources available.Where there are no resources available that meet the request's requiredattributes or constraints, the scheduler will respond with a failureresult 111.

If the scheduler is able to satisfy the request's attributes andconstraints, the scheduler produces implementation tasks 112, which arethen validated again by the rules engine 107 and once all issues areresolved sends the implementation tasks to the drivers 114, 115, 116,117 for technical implementation 113.

In an embodiment, the drivers 114, 115, 116, 117 are responsible forexecuting atomic actions, for example, provisioning a data volume on aspecified computer system. There can be multiple drivers, eachresponsible for implementing one or more atomic actions. Drivers caneither be on the same machine as the scheduler or on remote machines.The scheduler uses standard network protocols to send messages to thedrivers, and the scheduler will report the result 111 of executing theimplementation tasks 113.

The rules engine's inputs 101 with business objectives translators 105can be running on all computer systems participating in the distributedsystem. These different inputs may come through different protocols withdifferent encodings.

Business objectives 102 can be specified in a high-level, naturallanguage, as can commercial, compliance, or regulatory objectives 103.Other types of objectives may also be provided in a similar fashion. Anexample business objective might be “Production data is important andneeds to be always available.” A regulatory objective might be “Customerdata must be encrypted.”

Environmental constraints 104 can be specified in a similar fashion tobusiness objectives 102. Environmental constraints can be used torestrict placement based on environmental factors, such as resourceavailability.

In an embodiment, new or updated objectives or environmental constraintsare translated to rules 105 (see also FIG. 4) and then stored in therulebase 106.

In an embodiment, the rules engine 107 will re-evaluate the rulebaseagainst the current state and inform the scheduler if any changes to thecurrent state are required.

Scheduler

FIG. 3 illustrates an embodiment of the rules engine and a schedulerwhen used to provision data storage volumes. In this example embodiment,data storage volumes are being requested 301. Data storage volumes aresingle accessible storage areas that can be provisioned on a computersystem for storing application data. Storage volumes may be made up ofdisk drives or other storage media that are physically attached to thecomputer system, or made up of remote storage media that is attached tothe computer system over networks or other methods.

In an embodiment, the scheduler 306 receives the individual storagevolume requests as events 302, 303, and 305. The scheduler asks therules engine 307, that given the request details, what changes need tobe made to the environment and what attributes and constraints must besatisfied in order to complete the request.

In an embodiment, the scheduler will evaluate the change, theattributes, and the constraints against the running environment,including the available resources 314, and attempt to satisfy them inthe most efficient way possible.

In an example, Volume A 302 is being requested. Volume A might haveattributes such as size=10 GB, and application=database. When therequest is made to the scheduler 306, the scheduler sends a request tothe rules engine 307. The rules engine may have a rule that constrainsdatabases to run only on disk type=SSD, and returns that constraint tothe scheduler. The scheduler then uses this constraint to find thecomputer systems in Group A 309, that have disks of type=SSD. Thescheduler may then choose which of the two computer systems 308, 310 hasthe most resources free and then select that system for provisioning thedata volume.

Input Processing

FIG. 4 is an example of a flow diagram that describes the logicalcomponents in one embodiment. The example of FIG. 4 illustrates how thedifferent inputs are processed when adding new rules.

Components in the dynamic language based rules input providers 201provide semi-natural language input capabilities where users would beinteracting with the system through bots in computer-based chatapplications 202, emails by replying or sending emails with severalwords about desired rules 203, or other communication methods that areeasily accessible to non technical users 205.

In an embodiment, these inputs are then sent to lexical processing unit210 that parses the user's input and validates initial syntax. If thelexical processing unit cannot parse the input, the lexical processingunit informs the user about parsing error.

In an embodiment, each input interface can serve as a bidirectionalcommunication channel where a user can get current information aboutexisting rules.

Besides dynamic language input providers 201, rulebase 215 also acceptsrule create or update requests through standard application programminginterfaces 206, such as Hypertext Transfer Protocol (HTTP), RemoteProcedure Call (RPC), and gRPC (GitHub RPC). These input methods wouldtypically use structured input formats that do not require lexicalprocessing.

In an embodiment, a graphical user interface 207 provides a user withcertain tools such as autocomplete functionality, detailed descriptionsof selected rule features and can instantly detect dependent rules andinform the user about possible improvements in rule execution order.

In an embodiment, environmental constraints 104, 208 are a type of rulethat determines placement constraints based on the discoveredenvironment. For example, an environment scanning submodule 404 (FIG. 5)may determine that a computer system is not, or is no longer, suitablefor running a particular process. When an environmental constraint isadded or changed, existing rules must be re-evaluated 212, and anychanges to those rules must be updated in the rulebase 211.

In an embodiment, one of the key rulebase input providers is a domainspecific language extension 209 that provides a flexible way to describeresource specific rules. In one embodiment, the domain specific languageextension includes a database of storage implementation rules thatallows the rules engine to evaluate and codify business rules in termsof technical implementation operators for a storage system. The databasemay include functionality of the storage system and may allow thescheduler to manage the data plane of a storage system. The list oftechnical implementation operators includes but is not limited to: theprovision of a volume including the attributes of the volume such as thename and size of the volume of data; the data protection attributes of avolume such as the utilization of one or more encoding methods that addredundancy to the data within a volume, or encode data in a format wherethe data is recoverable if a given fraction of the data is no longeraccessible given the remaining fraction of the data; the dataavailability attributes of a volume such as the utilization of one ormore replication methods where a copy of the data is maintained on othercomputer or systems other than the system holding the original data; thedata availability attributes of a volume such as the service definitionsof the replication methods available to a storage system including, forexample, methods which may replicate data in real time ensuring that acopy of the data is always consistent with the source data as well asmethods where data in a copy is only eventually consistent; the datasecurity attributes of a volume such as the use of cryptographic methodswhich are implemented to scramble the data with a key such that the datais only later accessible by decoding with the same key; the datasecurity attributes of a volumes such as the use of access controlmechanisms to restrict access to the volume to specific computer usersor specific computer systems; the data reduction attributes of a volumesuch as the use of data compression methods to reduce the size of thedata that needs to be persisted; the data reduction attributes of avolume such as the use of the data deduplication methods to removeduplicate copies of the data; the data caching attributes of a volumewhere commonly used data within the volume is retained in a fastermedium, for example computer system memory or other fast storage medium,such that access to the data is faster than would be possible from thestorage medium where the data is persisted; the methods and technicalsystems that are available to access a given volume including, forexample, the formats and protocols used to access a volume through astorage system; the attributes of different classes of storage mediathat are available in a storage system including attributes such are thespeed of the media, the format and access protocol to the media, thedurability and expected lifetime of the media, and any constraints thatmay apply when accessing the media.

In addition, the database may include links of business operators to thetechnical operators of the storage system such that the rules engine canparse the business operators and identify the appropriate attributes andactions to be taken by the storage system. Examples of links betweenbusiness operators to technical operators may include but not limitedto:

“secure” to identify cryptographic methods;

“highly available” to identify replication methods;

“immediately available” to identify synchronous or highly consistentreplication;

“allow access” to identify security controls for access controlmechanisms;

“fast” to identify the caching methods used or the speed of the diskmedia where the volume is placed; and

“archive” to identify the type of disk media to select.

In another embodiment, other domain specific databases can be added bythe operator of the computer or storage system to allow the rules engineto process parameters for system functionality or computer systemresources that have not yet been defined. This allows the technique tobe extensible to cover the development of new technologies that have notbeen released as products yet.

In an embodiment, changes to the rulebase are queued in a pipeline 214,which ensures that changes are applied sequentially in the order thatthey arrive.

The rulebase 215 is a database that stores the processed rules.

Rulebase

FIG. 5 illustrates an embodiment of a rulebase state management system400 having a rulebase platform that can work with multiple sources ofdata and that constantly re-evaluates its own state. The rulebaseconsists of any number of rules stored in the rulebase 423 and anynumber of data provider submodules 422. When a rules engine computerprogram starts 401, it is responsible for initializing the rulebase andall of the data provider submodules. The rules engine is startedalongside the scheduler. The scheduler is responsible for actual taskimplementation. During startup procedures, the rules engine isresponsible for invalidating all existing rules stored in the rulebaseand rebuilding the rulebase from scratch. This is necessary since therules engine has no knowledge of how the environment or rules may havechanged during the shutdown period.

The current rulebase architecture allows for parallel bootstrapping 402of all data gathering submodules 422. Bootstrapping is the process ofstarting the system and performing any initialization tasks. Usingcomputer language specific features that enable creation of lightweightprocesses, the rules engine can efficiently collect data in anasynchronous way. These submodules can either be running inside the mainrules engine computer program or in a separate computer that is runningin the same datacenter, or a different datacenter in a differentgeographical location.

In an embodiment, database initialization 403 is required to enablerulebase persistence. There is no restriction on database type forstoring rules, however, since rules are dynamic, a document-based orkey/value type database with a flexible data schema is preferred. Eachrule is assigned a unique identifier that is then used to update ordelete specific rules. This unique identifier is used as a key in thedatabase and the rule payload is the value.

In an embodiment, the environment scanning submodule 404 is responsiblefor gathering data about the current environment in which the computerprogram is running. There can be multiple different environments such asbare-metal servers (e.g., operating systems are running directly onhardware), virtual machines (e.g., operating systems are running onhypervisors), and containers (e.g., applications running in sandboxedenvironments). There can be more types of environments and theenvironment scanning submodule 404 is responsible for identifying theseenvironments. Environment information is collected and sent as state forlater use by the rules engine.

The environment scanning submodule 404 may also be able to applyunstructured heuristics to capture key variables like disk speed andcapacity. These heuristics and variables do not need to be pre-definedand can be learned from the environment or system being scanned. Theheuristics may, by way of example, set a variable defining a diskresource as “fast” if it determines it is fast in comparison to otherdisks in the system. This variable could later be used to satisfy arequest constraint such as “Databases should be on the fastest disks.”

The business requirement analysis submodule 405 is responsible forproducing business related rules. By analyzing multiple channels such astext based input systems, direct API, graphical user interface, domainspecific language inputs or any other data source, it can then createrules that influence scheduler behavior. This influence can includeplacing boundaries on available resources to reduce costs, ensureasynchronous replication to slower computer data disks while preservingfast computer data disks (which are usually more expensive) to activeworkload.

The number of active submodules is not limited. As long as separatesubmodules implement a common communication protocol they are allowed tojoin the rulebase program and provide either new rules, rule updates orinform about state changes.

In an embodiment, a compliance requirements submodule 407 is responsiblefor producing compliance related rules such as data locality, archivesfor a specified period of time (to enable auditing). These rules couldthen be grouped into layers, moving data into cheaper storage tiers.Typically, this is a manual process that requires many man-hours toidentify data required by compliance and then migrating it into cheaperdata storage systems.

Rule initialization 408 is a step implemented to validate rules thatwere stored inside the rules engine database. Since the rules enginecomputer program version can be updated, rule interpretation can bechanged. In this step, it is also evaluating the current environmentwhich is present at initialization time.

In an embodiment, every submodule 409, 410, 411 of the rules engine hasto provide data to the rules engine state. The rules engine has multipledifferent data ingress interfaces which provide appropriate access todifferent submodules.

The environment scanning module 404 runs inside the rules enginecomputer program and therefore handles data updates internally in a veryperformant way.

The business requirement analysis submodule 405 can be running on everycomputer system in a distributed system and therefore uses protocolsthat are designed for remote communications to provide data updates tothe rules engine state 419.

The compliance requirements submodule 407 runs in a similar way as thebusiness requirement analysis submodule 405 and uses protocols designedfor remote communication. Each submodule is responsible for datadurability and therefore has to perform retries on failure.

A data aggregation point 421 is responsible for unifying data formats.Multiple submodules can send data in different protocols or encodings.This step is needed to read and decode data from incoming connectionsand save it into the rules engine state 419.

A monitoring step 412 allows reaction to rules or environment changes.This step is critical to build a complex rules engine with multiplelevels of inter-dependent rules. The monitoring step mayopportunistically evaluate to ensure optimal placement and policyenforcement across the environment.

A checking step 413 allows the rules engine, depending on the databaseimplementation of the rules engine state, to react to changes to theenvironment or rules. If the state database provides a subscriptionmechanism to get notification of data changes, then the rules engineuses this mechanism to subscribe to change events, otherwise (if thedatabase doesn't support this functionality), a separate process threadis created that constantly queries the database and checks for changes.

A dependent rules identification step 414 is performed when rules changeor during the rules engine initialization process. In this step, therules engine is checking which rules might be dependent on other rules.As an example, there could be two rules: encryption and compression. Ifa system would try to compress already encrypted data it would fail toreduce the size due to the nature of encryption, therefore, datacompression should be done before encryption to achieve desired results.

A rules validation step 415 is performed after identifying dependentrules. This step is needed to check whether the rule is valid. Rulesyntax checking is performed in the submodules and then at aggregationpoint 421. This step is required for deeper analysis of the rule wheredata gathered from dependent rules can be combined with the currentenvironment in which the system is running. An example for this step canbe a data replication rule that enforces replicating a data volume toanother geographical location. The rules engine can then check dataprovided by the environment scanning submodule 404 to verify that thesystem has at least two geographical locations and therefore replicationto another geographical region is feasible. If this verification resultsin a negative outcome, this rule and its dependent rules will be markedas inactive or it will prohibit new data volume creation depending onsupplied policies. This functionality can strictly enforce businessrequirements and reduce damage done by outages, wrong user decisions, orother unknown factors.

An execution order creation step 416 is the next step after identifyingdependent rules 414 and validating rules 415. In this step, the rulesengine analyzes the rule's dependency tree and optimizes execution orderbased on predefined templates. These templates can either be strict, forexample, backup data only after compression or an algorithm in a form ofsource code can be supplied to resolve execution order. For example, ascript written in a popular programming language such as Python can besupplied, which will be supplied with the rules and environment, and theoutput data of executing that script will be used to resolvedependencies. This functionality allows complex dependency resolutionlogic to be added at runtime without introducing business relatedcomplexity into the main source code of the rules engine, sincedifferent organizations have different problems and different solutionsto those problems. This process starts the apply rules step 417 and theupdate rulebase step 422, which run in parallel.

The apply rules step 417 is a process created by the rules engine thatwill trigger state evaluation. The apply rules step is needed to computerequired updates (if any required is determined by the rulesthemselves).

The update rulebase step 422 is a step in the rules engine statemanagement where the system is storing rules that were formatted, theirdependencies identified, and the execution order created. This is thefinal stage of adding or updating rules.

An update state step 418 is a step that affects rules engine state. Inthis step, the system is calling the state database directly, sinceexisting rules were updated and therefore triggering a new state changeevent that may result in actions taken by the scheduler (the rulesengine state 419 is also shared with the scheduler).

In an embodiment, the rules engine state 419 is a shared database thatis being used by the rules engine and the scheduler. The rules enginestate is constantly being updated by various other submodules, thescheduler, and the rules engine itself. This database has to providefast access to data. For example, a node that has an active rules engineand a scheduler process can keep a copy of the shared state in memoryfor performance gains. Typically, this data can be recreated fromscratch by the data gathering submodules in a case of system failure.Distributed, eventual consistency type database is recommended toprovide faster recovery in a case of node failure.

The rules engine database (rulebase 423) is the database which is usedto store rules. All rules that are added to this database have to bepreprocessed, it has no logic to do any changes to the rules. Typically,this database needs high availability.

Identifying Dependent Rules

FIG. 6 is an example that illustrates how the rules engine resolvesdependencies between rules. At block 101, when a new rule is added tothe rules engine, a check is performed at block 702 against existingrules, and the check is determining whether the new rule might directlyor indirectly affect other already stored rules.

At decision point 703, if during this check the rules engine determinesthat no rules can be affected by this new addition, the rules engineskips to the last step, block 706, and saves the rule to the rulebasedatabase 707.

If during the check at block 702, any dependent rules are identified,the rules engine proceeds to start a dependency investigation at block704. During the investigation, the rules engine is trying to identifyhow to improve the current rulebase configuration. For example, thisinvestigation can include checking predefined best practices for rules,such as choosing highest data compression level for backup storage, andselecting correct order for rules such as by ensuring compression beforedata encryption (data encryption is a process of scrambling data so onlyauthorized parties can decrypt data and read it).

After rule investigation, at block 705 the rules engine updates thecurrent rule and groups dependent rules so later it is faster toidentify which rule is affecting other rules. Once updates are finished,at block 706, the rules engine saves changes to the rulebase database.

Environment Scanning Submodule

FIG. 7 illustrates an example of the environment scanning submodule 800of the rules engine. Submodules are required to provide additionalcontext to the rules engine. There may be many types of submodules thatcollect data about different types of computer systems. The computersystem that a submodule is scanning is not necessarily a singlecomputer, it can be a cluster of computers controlled by a third partyplatform. Submodules can be running either in the environment oroutside, and submodules support different types of configurations.

When an environment scanning submodule starts, 801, it begins byidentifying the environment 802. In an embodiment, environmentidentification process consists of several parallel 803 tasks.

The first task is identifying the platform itself 804. There are anincreasing number of platforms available today and these platformsprovide easier ways to manage large numbers of computers. Identifyingthe platform is an important step since having information about theplatform details will help the rules engine correctly configure itsdrivers. Platform identification can be done in several ways. One of theeasiest options involves looking for environment variables supplied tothe process or looking for a configuration file on the computer harddisk.

Another task in the environment scanning module is identifyingprivileges 805. Depending on deployment and user preferences, the rulesengine can either be granted or not certain privileges, for example,access to resource quotas, limits, and the right to move workloads fromone computer to another. All this information has to be gathered andsent to the rules engine interface, then the correct driver interpretsthe privilege information.

Storage capabilities identification 806 depends on the system that theprogram is running in. For example, storage capabilities when the systemis running in a platform that manages a cluster of computers and hasrights to attach remote storage, capabilities can be either increased ordecreased on demand. If a system is running on a single computer,usually storage capabilities are those provided by the hard disksattached to computer.

Network capabilities identification 807 scans for availability ofrequired ports. Since the rules engine and the whole system that isusing the rules engine has certain network requirements, this submodulehas to ensure that those requirements are met. Some systems provideoptions to dynamically update network configuration, this information isalso relevant to the rules engine.

Pluggable arbitrary code execution middleware 808 is one of the novelways to gracefully deal with the custom platforms on which a rulesengine is deployed. This part of the environment scanning submodule cansafely execute custom source code that can be written specifically forthe targeted system to gather the most detailed information about theenvironment. One of the example uses cases can be identifying customplatform capabilities where only company that operates it knows how toget required information, therefore a custom code could be written thatwould constantly be used to check current system state.

An aggregation point 809 in the environment scanning submodule 800 iswhere the application is combining data from multiple sources into asingle format, then optionally encodes 810 (e.g., compresses) the datafor network transport and sends the data to the rules engine ingressinterface 811.

Rules Engine Behaviour in a Cluster

FIG. 8 illustrates an example of how the rules engine works in a cluster900 of six computers (e.g., computer nodes 901-906). This number ofcomputers was chosen only to illustrate behavior and this system is notlimited to use this number of computer nodes. If needed, the cluster canhave more than a thousand computer nodes. This diagram also includes thescheduler component which is running together with the rules engine anduses the rules engine to verify implementation plans or to get eventsfrom the rules engine that could trigger computer resource reschedulingor any other action that the scheduler is capable of implementing. Thecluster also includes a computer node 907 that stores a database.

In the embodiment of FIG. 8, one computer node 901 is different from theother computer nodes 902-906 in a way that only node 901 has an activescheduler and rules engine. All the computer nodes 902-906 that areadded to the same cluster are participating in a consensus algorithmthat results in electing a leader node.

Once a leader node is elected, the leader node activates the rulesengine and the scheduler. Other nodes are then waiting for the nextleader election which will be triggered once the current leader resignsor goes offline. Since there is only one active scheduler in a clusterat a time, and only the scheduler accesses the rules engine, the rulesengine does not need to be active on multiple nodes concurrently.

Each computer node (e.g., server) also runs data gathering submodulesthat are responsible for keeping up-to-date state in the rules engine.The rules engine has to be notified about computer cluster changes(e.g., nodes joining and leaving), hardware changes, and availableresources in geographical zones. In the example of FIG. 8, only theactive rules engine from computer A 901 has write access to database 907and is saving all data gathered from the cluster. This technique forupdating computer cluster state from a single node that is being electedby the rest of the computers through a consensus algorithm avoids datarace conditions (data race is a term used to describe when two processesare trying to manipulate the same piece of information at the same timeand are not aware of each other, this can result in corrupted data).This technique also serves as a way to detect node failures through anindirect route, e.g., missing updates. When a scheduler or rules enginedetects that the continuous stream has stopped, an investigation taskcan be started to detect a cause for the stop (e.g., node failure,network failure or it could have happened due to multiple otherreasons), nor rules engine nor scheduler is performing thisinvestigation task.

The database 907 used by the rules engine can be of any type, however, aschema-less database is preferred due to the dynamic nature of gathereddata. A schema-less database does not impose a structure on the type ofdata that is stored within it, allowing new data types to be storedwithout database reconfiguration. The database is used to store rulesand state sent by the rules engine.

Multidimensional Rules

FIG. 9 illustrates an example of how multi-dimensional rules influencedata volume provisioning and continuously work on enforcing rules.

Multidimensional rules allow the rules engine to effectively process aset of rules that have a non-linear relationship and that eitherconflict or provide synergy with other rules.

An example of conflicting rules is one rule that enforces the cheapestresource utilization and another that enforces highest performance.Since each rule, run individually, would produce a different result,when used in combination the result is unpredictable. If cost wasprioritized, then on the next evaluation, performance criteria wouldfail and an action might be triggered to re-schedule the resource.

In order to solve this issue, multidimensional rules allow the detectionof dependent rules and any conflicts or synergies in multiple measurabledimensions (i.e., cost and performance).

Where a conflict or synergy is found, the rules engine starts performingtrials. A trial is an operation when the rules engine subjectivelydecides which rule takes precedence over another rule or rules andinstructs the scheduler to deploy a resource in favor of the first rule.So, with reference to FIG. 9, once a data volume resource creation isrequested 501, a scheduler asks the rules engine 502 for policiesdefined for this particular volume. In this case, the rules enginedetects multiple rules 503 and based on priority (or randomly if nopriority is set) decides according to which rule the volume should bedeployed 504 (e.g., placed on a particular computing resource).

Once a resource has been deployed (e.g., placed on a particularcomputing resource) the rules engine starts collecting performancemetrics 505. For example, performance metrics may relate to: the speedof memory access (e.g., latency), the number of input/output (I/O)operations per unit of time, the size of I/O operations, replicationlatency, the size of a queue, CPU utilization, or any combinationthereof. The rules engine then generates a score 506 for this deploymentbased on at least one of the performance metrics. For example, the rulesengine may compute an average score for a particular performance metricor average scores for multiple performance metrics. The score may besome value that corresponds to one or more of the performance metricsthat are collected.

If this score (or scores) is deemed unacceptable 507, the rules engineis notified of the score 508. The score may be evaluated against manycriteria to determine whether the score is acceptable. For example, thescore may be deemed acceptable if the score meets the average score ofother resources in the cluster, or if it meets a user-definedrequirement. The criteria for judging the acceptability of a score canbe pre-loaded into the rules engine. Additionally, the criteria forjudging the acceptability of a score may be application-specific,customer-specific, and/or dependent on some other factor or factors.

If the score is not deemed to be acceptable, the rules engine 502 canadjust rule attributes and try a different configuration for therequested volume, instructing the scheduler to change the deployment ifnecessary.

In an embodiment, rules are constantly re-evaluated and the processrepeats (e.g. in an iterative manner) until an acceptable score or untilthe best score has been achieved. In an embodiment, the best score maybe the highest score that is found during some number of iterations.

Some rules might enforce resource redeployment only during hours whenthe resource is not used (or during hours of lower or minimum activity(e.g., relatively low I/O operations), e.g., at a time when thecorresponding enterprise is closed for business). Implementing resourceredeployment on during hours when the resource is not used (or duringhours of lower or minimum activity, e.g., at a time when thecorresponding enterprise is closed for business), allows for nondisruptive automated cluster tuning.

Once the score is deemed acceptable or the best score has been achievedwith all other constraints met, the rules engine marks deployment assuccessful 509.

Object Descriptors

Object descriptors are strings of information that describe resourcesthat are processed by the rules engine and the scheduler. The objectdescriptors can be attached to a resource by an operator, or can beautomatically inferred by the rules engine upon evaluation of therulebase. The object descriptor may also be attached by an environmentscanning submodule 404 (FIG. 5).

The rules engine may evaluate one or more object descriptors to deriveattributes or environmental constraints.

Various techniques for storing a volume of data are discloses.

In an embodiment, a rules processing method and system that includes alexical processing unit such that business objectives can be interpretedinto a rule-based format suitable for application by a computer systemis disclosed.

In an embodiment, the technique involves a variety of natural languageinput methods.

In an embodiment, the technique involves a rules engine.

In an embodiment, the technique involves a rules engine that can adaptone or more running computer systems.

In an embodiment, a scheduler can process events and operator requestsusing the parsing capability of the rules engine to determine attributesand environmental constraints of a computer system or a number ofcomputer systems.

In an embodiment, a scheduler implements technical operators and affectschanges to the configuration of a computer system or a number ofcomputer systems based on events and operator requests.

In an embodiment, the technique involves a method to process businessintent through the definition of business operators which are linked totechnical operators. Technical operators can be implemented within acomputer system to implement the business intent described in thebusiness operators.

In an embodiment, the technique further involves a database thatdescribes technical operators relating to a storage system such thatattributes of volumes and technical implementations of a storage systemcan be codified and linked to business operators.

In an embodiment, the technique further involves a scheduler that canimplement technical operators and affect changes to the configuration orrunning operation of a computer system of a number of computer systemswhere such computer systems provide resources to a storage system.

In an embodiment, the technique further involves a scheduler that canimplement technical operators and affect changes to the configuration orrunning operation of a computer system of a number of computer systemswhere such computer systems utilize resources from a storage system.

In an embodiment, the technique further involves a scheduler that canimplement technical operators and affect changes to the configuration orrunning operation of a computer system of a number of computer systemswhere such computer systems are components and resource of a storagesystem.

In an embodiment, additional domain specific language databases can beadded to the system to allow the rules engine and scheduler to manageother technologies as they are added to an existing environment.

In an embodiment, the technique involves a method where operatorsupplied computer code is integrated into the rules parsing engine toallow the rules engine to determine attributes from the computer systemenvironment that are propriety or not commonly known to the rules engine

In an embodiment, the technique involves a method where operatorsupplied computer code is integrated into the scheduler to allow thescheduler to apply changes to a computer system or a number of computersystems where those systems are propriety or not commonly known to thescheduler.

In an embodiment, operator supplied computer code is integrated into thescheduler to allow the scheduler to implement technical operators andaffect changes to the configuration or running operation of a computersystem of a number of computer systems where such computer systemsutilize resources from a storage system.

In an embodiment, a multi-dimensional rules engine can work with nonlinear relationship based requirements and tune a working system throughexperimentation and performance measurement.

As used herein, a computer application or application program may be asoftware program that runs on a computer and is the most common softwareon computers. Web browsers, e-mail programs, word processors, anddatabases are all example of computer applications.

As used herein, a file may be a collection of digital data stored in oneunit, identified by a filename. The file can be a document, picture,audio or video stream, a data library, application, or anothercollection of data.

As used herein, a database may be a data structure that storesinformation and data that is organized to allow easy retrieval of theinformation. Typical databases contain multiple structures calledtables, which may each include several different fields of data. Forexample, a company database may include tables for products, employees,and financial records. Each of the tables may have different fields thatare relevant to the information stored in the table.

As used herein, a storage system may be a collection of computers,software applications, and storage devices that together operate toprovide the persistence and retrieval of data from storage devices toand from computer applications and databases.

As used herein, a data block, a block of data, or simply a “block,” is asequence of bytes or bits, typically containing some whole number ofrecords, that has a maximum length, e.g., a block size. The process ofputting data into blocks is called blocking, while deblocking is theprocess of extracting data from blocks. Blocked data is normally storedin a data buffer and read or written a whole block at a time. Blockingcan reduce the overhead in storage systems and can speed up the handlingof a data-stream. Blocking is usually implemented when storing data to9-track magnetic tape, to NAND flash memory, and to rotating media suchas hard disks and optical disks. Most file systems are based on a blockdevice, which provides a level of abstraction for the hardwareresponsible for storing and retrieving specified blocks of data, thoughthe block size in file systems may be a multiple of the physical blocksize. Block storage is typically abstracted by the file system ordatabase management system (DBMS) for use by computer applications andend users.

As used herein, a volume of data is a single accessible storage areawith a single file system, typically, though not necessarily, residingon a single partition of a hard disk. In an embodiment, a volume of datacan be different from a physical disk drive, however, the volume of datacan be accessed with an operating system's logical interface. A volumeof data is a logical representation of a number of data blocks, whichdata blocks are concatenated to form a larger set of data than can bestored as a group of data blocks. A volume of data is not the same as apartition in computer storage. For example, a floppy disk might beaccessible as a volume, even though the floppy disk does not contain apartition, as floppy disks cannot be partitioned with most moderncomputer software. Furthermore, an operating system can recognize apartition without recognizing any volume associated with the partition,as when an operating system cannot interpret the filesystem storedthere. Volumes of data exist at the logical operating system level whilepartitions exist at the physical, media specific level. Sometimes thereis a one-to-one correspondence, but this is not a requirement. InUnix-like operating systems, volumes other than the boot volume have amount-point somewhere within the filesystem, represented by a path.Logically, the directory tree stored on the volume is grafted in at themountpoint. By convention, mount-points will often be placed in adirectory called ‘/mnt’, though ‘/media’ and other terms are sometimesused. Files within a volume of data can generally be moved to any otherplace within that volume by manipulating the filesystem, without movingthe actual data. However, if a file is to be moved outside the volume,the data itself must be relocated, which is a much more expensiveoperation.

Although the operations of the method(s) herein are shown and describedin a particular order, the order of the operations of each method may bealtered so that certain operations may be performed in an inverse orderor so that certain operations may be performed, at least in part,concurrently with other operations. In another embodiment, instructionsor sub-operations of distinct operations may be implemented in anintermittent and/or alternating manner.

It should also be noted that at least some of the operations for themethods may be implemented using software instructions stored on anon-transitory computer useable storage medium for execution by acomputer. As an example, an embodiment of a computer program productincludes a computer useable storage medium to store a computer readableprogram that, when executed on a computer, causes the computer toperform operations, as described herein.

Furthermore, embodiments of at least portions of the invention can takethe form of a computer program product accessible from a computer-usableor non-transitory computer-readable medium providing computer executableinstructions, or program code, for use by or in connection with acomputer or any instruction execution system. For the purposes of thisdescription, a non-transitory computer-usable or computer readablemedium can be any apparatus that can contain or store the program foruse by or in connection with the instruction execution system,apparatus, or device.

The computer-useable or computer-readable medium can be an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system(or apparatus or device). Examples of a computer-readable medium includea semiconductor or solid state memory, magnetic tape, a removablecomputer diskette, a random access memory (RAM), a read-only memory(ROM), a rigid magnetic disk, and an optical disk. Current examples ofoptical disks include a compact disk with read only memory (CD-ROM), acompact disk with read/write (CD-R/W), and a digital video disk (DVD).

In an embodiment, the above-described functionality is performed atleast in part by a computer or computers, which executes computerreadable instructions. FIG. 10 depicts a computer 1000 that includes aprocessor 1002, memory 1004, and a communications interface 1006. Theprocessor may include a multifunction processor and/or anapplication-specific processor. Examples of processors include thePowerPC™ family of processors by IBM and the x86 family of processors byIntel such as the Xeon™ family of processors and the Intel X5650processor. The memory within the computer may include, for example, anon-transitory storage medium such as read only memory (ROM), flashmemory, RAM, and a large capacity permanent storage device such as ahard disk drive. The communications interface enables communicationswith other computers via, for example, the Internet Protocol (IP). Thecomputer executes computer readable instructions stored in the storagemedium to implement various tasks as described above.

In the above description, specific details of various embodiments areprovided. However, some embodiments may be practiced with less than allof these specific details. In other instances, certain methods,procedures, components, structures, and/or functions are described in nomore detail than to enable the various embodiments of the invention, forthe sake of brevity and clarity.

Although specific embodiments of the invention have been described andillustrated, the invention is not to be limited to the specific forms orarrangements of parts so described and illustrated. The scope of theinvention is to be defined by the claims appended hereto and theirequivalents.

In the above description, specific details of various embodiments areprovided. However, some embodiments may be practiced with less than allof these specific details. In other instances, certain methods,procedures, components, structures, and/or functions are described in nomore detail than to enable the various embodiments of the invention, forthe sake of brevity and clarity.

Although specific embodiments of the invention have been described andillustrated, the invention is not to be limited to the specific forms orarrangements of parts so described and illustrated. The scope of theinvention is to be defined by the claims appended hereto and theirequivalents.

What is claimed is:
 1. A method for provisioning a volume of data, themethod comprising: identifying a set of rules associated with the volumeof data, wherein the set of rules includes at least two conflictingrules; prioritizing the at least two conflicting rules; placing thevolume of data on a first computer system according to theprioritization; collecting performance metrics corresponding to theplaced volume of data; computing a score based on the collectedperformance metrics; determining if the computed score is acceptable;determining that the placement is successful if the computer score isdetermined to be acceptable; and changing the prioritization of the atleast two conflicting rules and placing the volume of data on a secondcomputer system according to the changed prioritization if the score isdetermined to be unacceptable.
 2. The method of claim 1 wherein thevolume of data is placed on the second computer system only during hourswhen the second computer system is not used.
 3. The method of claim 1wherein the volume of data is placed on the second computer systemduring a period of time in which activity is relatively low.
 4. Anon-transitory computer readable medium that stores computer executablecode, which when executed by one or more processors, implements a methodfor provisioning a volume of data, the method comprising: identifying aset of rules associated with the volume of data, wherein the set ofrules includes at least two conflicting rules; prioritizing the at leasttwo conflicting rules; placing the volume of data on a first computersystem according to the prioritization; collecting performance metricscorresponding to the placed volume of data; computing a score based onthe collected performance metrics; determining if the computed score isacceptable; determining that the placement is successful if the computerscore is determined to be acceptable; and changing the prioritization ofthe at least two conflicting rules and placing the volume of data on asecond computer system according to the changed prioritization if thescore is determined to be unacceptable.
 5. The non-transitory computerreadable medium of claim 4, wherein the volume of data is placed on thesecond computer system only during hours when the second computer systemis not used.
 6. The non-transitory computer readable medium of claim 4,wherein the volume of data is placed on the second computer systemduring a period of time in which activity is relatively low.